Image processing device and method for processing image

ABSTRACT

An image processing device includes: a memory; and a processor coupled to the memory and configured to: obtain image data regarding an image captured by an image capture device that moves along with a moving body and of which image capture direction is a certain traveling direction of the moving body, and detect a feature caused by a reflection from features based on information regarding positions of the features in the image data at a time when the moving body has moved in a direction different from the image capture direction.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromthe prior Japanese Patent Application No. 2012-255936, filed on Nov. 22,2012, the entire contents of which are incorporated herein by reference.

FIELD

The technique disclosed herein is related to a technique for processingan image.

BACKGROUND

In these years, various techniques are being developed in order to helpdrivers drive safely. One of the techniques is a technique forrecognizing a recognition target in an image captured by a cameramounted on a vehicle. When a recognition target has been recognized, awarning or the like is issued to a driver.

For example, when the camera is installed inside the vehicle, the cameracaptures images of the outside through a sheet of glass. In this case,so-called “reflections” might occur in an image captured in anenvironment in which an intense light source such as the sun exists.Reflections are phenomena in which light from objects inside the vehicleis reflected by a surface of the sheet of glass or the like and detectedby a sensor of the camera installed inside the vehicle, thereby causinga captured image to include images of the entirety or part of theobjects inside the vehicle. Reflections occur when there is an objectwhose surface reflects light, such as a sheet of glass, in an imagecapture direction of the camera as in the case of the camera installedinside the vehicle. In the following description, an image includingreflections will be referred to as a reflection image.

When a recognition target is to be recognized in a reflection image,recognition accuracy decreases. Therefore, an object recognition devicefor properly recognizing a recognition target even in a reflection imagehas been disclosed. The object recognition device is disclosed inJapanese Laid-open Patent Publication No. 2010-61375. The objectrecognition device sequentially captures images of a region in a certaindirection of a vehicle and extracts a plurality of feature points fromeach image. The object recognition device then removes fixed featurepoints whose coordinates remain the same in the images from a pluralityof feature points extracted from a latest image. Finally, the objectrecognition device recognizes an object on the basis of the featurepoints in the latest image from which the fixed feature points have beenremoved.

SUMMARY

According to an aspect of the invention, an image processing deviceincludes: a memory; and a processor coupled to the memory and configuredto: obtain image data regarding an image captured by an image capturedevice that moves along with a moving body and of which image capturedirection is a certain traveling direction of the moving body, anddetect a feature caused by a reflection from features based oninformation regarding positions of the features in the image data at atime when the moving body has moved in a direction different from theimage capture direction.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are diagrams illustrating a reflection;

FIG. 2 is a functional block diagram illustrating a driving supportapparatus including an image processing device according to anembodiment;

FIG. 3 is a diagram illustrating an example of accumulated information;

FIG. 4 is a diagram illustrating a process for accumulating features;

FIG. 5 is a diagram illustrating a process for checking features;

FIGS. 6A and 6B are flowcharts illustrating image processing;

FIG. 7 is a flowchart illustrating the process for accumulatingfeatures;

FIG. 8 is a flowchart illustrating the process for checking features;

FIG. 9 is a flowchart illustrating a process for detecting reflections;and

FIG. 10 is a diagram illustrating an example of the hardwareconfiguration of the image processing device.

DESCRIPTION OF EMBODIMENTS

The object recognition device that focuses upon a fact that featurepoints caused by reflections remain at the same coordinates insequentially captured images may remove fixed feature points from thesequentially captured images.

However, the accuracy of the object recognition device is insufficientto remove reflections. That is, it is difficult for the objectrecognition device to accurately detect reflections.

Therefore, an object of the technique disclosed in the embodiments is toaccurately detect reflections.

Embodiments will be described in detail hereinafter. The embodiments maybe combined with each other insofar as the content of processing doesnot cause a contradiction.

Generally, features caused by reflections remain at the same positionsin sequentially captured images. However, the inventors have focusedupon a fact that some features that have not been caused by reflectionsalso remain at the same positions in a plurality of images.

For example, features caused by an object located parallel to atraveling direction of a moving body remain at the same positions in aplurality of images. When the moving body is a vehicle and the travelingdirection is a forward direction of the vehicle, for example, suchfeatures may be features caused by lines configuring a traffic lane, aguardrail, or the like. That is, the above-described object recognitiondevice undesirably recognizes features that have not been caused byreflections as fixed feature points and accordingly removes thefeatures.

In addition, insufficient accuracy of detecting reflections results in adecrease in the accuracy of recognizing objects. For example, if objectrecognition is performed in an image from which the features of anobject to be recognized have been removed as fixed feature points, theobject recognition device does not recognize the object.

Therefore, in an embodiment that will be described hereinafter,reflections are accurately detected by distinguishing features caused bythe reflections and features caused by objects other than thereflections from each other more accurately.

FIGS. 1A and 1B are diagrams illustrating a reflection. FIG. 1A is adiagram illustrating the positional relationship between an imagecapture device mounted on a moving body, an object that causes areflection, and a sheet of glass that reflects light from the object. Inthis embodiment, the moving body is a vehicle 1. In this embodiment, thevehicle 1 includes an image capture device 11 therein. The image capturedevice 11 is, for example, a camera.

The image capture device 11 determines a direction including a certaintraveling direction of the vehicle 1 as an image capture direction, andsequentially obtains images in the image capture direction. In thisembodiment, the image capture device 11 is installed such that anoptical axis thereof is directed in the forward and downward directionsof the vehicle 1, in order to capture images of a surface of a road anda sight in the forward direction of the vehicle 1. That is, in thisembodiment, a direction including the forward direction of the vehicle 1is determined as the image capture direction.

However, the direction of the optical axis does not have to perfectlymatch the forward direction of the vehicle 1. That is, the image capturedirection of the image capture device 11 does not have to perfectlymatch the certain traveling direction of the vehicle 1 in terms of theangle, and the image capture direction may be a direction in which theimages of the surface of a road and a sight in the certain travelingdirection may be captured.

When the image capture direction is the forward direction, the sheet ofglass that causes reflections corresponds, for example, to a windshield2 of the vehicle 1. Here, when the image capture device 11 and theobject 3 exist on the same side relative to the windshield 2, the object3 is reflected. For example, this holds true for a case in which theimage capture device 11 and the object 3 exist in the vehicle 1. FIG. 1Billustrates an example of an image in which a reflection is occurring.

The image includes an outside region 100 captured through the windshield2. The outside region 100, however, includes an image of a reflection101 that does not actually exist outside the vehicle 1. The reflection101 is part of the object 3 reflected onto the windshield 2 in the formof a line due to sunlight reflected from an edge 104 of the object 3 (animage 3′ in the captured image).

Here, the position of the reflection 101 in the captured image isdetermined on the basis of the positional relationship between a lightsource and the object 3. Since the light source is the sun, the lightsource is located at infinity. Therefore, insofar as the object 3 doesnot move, the position of the reflection 101 does not change in a shortperiod of time relative to the movement of the sun even if the vehicle 1moves.

On the other hand, there are images of a line 102 configuring a trafficlane and a guardrail 103. In the following description, the line 102configuring a traffic lane will be simply referred to as a “lane line”.While the vehicle 1 is moving parallel to the image capture direction ofthe image capture device 11, features caused by the lane line 102 remainat the same positions in images even if the vehicle 1 moves forward. Thesame holds true for the guardrail 103. While the lane line and guardrailare described herein as an example, the present invention is not limitedidentifying any particular type of objects and maybe be used to detectand identify objects relative to safety of a moving body.

There is a warning device that warns a driver of deviation of a vehiclefrom a traffic lane in order to avoid an accident caused by thedeviation. The warning device is a device having a so-called lanedeviation warning function. When the vehicle is about to deviate from atraffic lane, the warning device calls the driver's attention by issuinga warning on the basis of the relative distance between the vehicle andthe traffic lane.

When the object recognition device disclosed in Japanese Laid-openPatent Publication No. 2010-61375 has been applied to the warningdevice, feature points caused by a lane line might be removed sincefixed feature points are deleted from a latest image. Therefore, becauseit is difficult to detect the lane line in an image from which thefeatures caused by the lane line have been removed, the warning devicedoes not appropriately issue a warning.

Therefore, an image processing device disclosed in this embodimentdistinguishes the reflection 101 from the lane line 102, the guardrail103, and the like. The image processing device then generates an imagefrom which the reflection 101 has been removed. Next, the imageprocessing device calculates the relative distance between the vehicle 1and the traffic lane on the basis of the image from which the reflection101 has been appropriately removed, in order to output the relativedistance to the warning device. The image processing device then outputsthe relative distance between the vehicle 1 and the traffic lane to thewarning device. The technique disclosed herein is not limited to this,and the image processing device may output the image from which thereflection 101 has been removed to another device, instead.

FIG. 2 is a functional block diagram illustrating a driving supportapparatus including the image processing device according to thisembodiment. A driving support apparatus 10 includes an image capturedevice 11, an image processing device 12, and a warning device 19. Theimage capture device 11 sequentially captures images of a region in theimage capture direction. Furthermore, the image capture device 11sequentially outputs image data regarding each of the plurality ofsequentially captured images to the image processing device 12.

The image processing device 12 executes a process for removingreflections. In this embodiment, the image processing device 12 alsoexecutes a process for recognizing a lane line and a process forcalculating the relative distance between the vehicle and the trafficlane. Details of the processes will be described later. The imageprocessing device 12 then outputs the relative distance to the warningdevice 19. The warning device 19 issues a warning to a driver on thebasis of the relative distance.

The image processing device 12 will be described more specifically. Theimage processing device 12 includes an obtaining unit 13, adetermination unit 14, a detection unit 15, a recognition unit 16, acalculation unit 17, and a storage unit 18.

The obtaining unit 13 sequentially obtains image data from the imagecapture device 11. The obtaining unit 13 then calculates the featurevalue of each pixel on the basis of the obtained image data. In thisembodiment, when calculating edge strength as the feature value of eachpixel, the obtaining unit 13 executes a process for extracting edges byapplying a known algorithm.

The determination unit 14 determines whether or not there has beenmovement in a direction different from the image capture direction. Thatis, the determination unit 14 determines whether or not there has beenmovement in a direction different from the certain traveling directionof the vehicle 1. Since the image capture direction is the forwarddirection of the vehicle 1 in this embodiment, the determination unit 14determines whether or not there has been movement in a lateral directionof the vehicle 1. The determination unit 14 determines whether or notthere has been lateral movement at a certain point of time using arelative distance calculated in the process for calculating the relativedistance executed in the past, details of which will be described later.Alternatively, the determination unit 14 may determine whether or notthere has been lateral movement in accordance with an output of asteering angle sensor, a vehicle speed sensor, a tire angle sensor, ayaw rate sensor, or the like.

The detection unit 15 detects a reflection on the basis of informationregarding the positions of features in an image at a time when it hasbeen determined that there has been movement in a direction differentfrom the image capture direction. For example, the detection unit 15detects features in an image at a time when it has been determined thatthere has been lateral movement. The detection unit 15 determines thatthere is a feature when the feature value of each pixel is equal to orlarger than a threshold.

The detection unit 15 then accumulates information. The accumulatedinformation is information that depends on the frequency of appearancesof a feature at each position of an image to be processed. For example,the accumulated information includes positional information andappearance strength. The appearance strength becomes larger as thefrequency of appearances of a feature at a certain position becomeshigher. That is, large appearance strength indicates that a feature hasfrequently appeared at the same position in images at times when it hasbeen determined that there has been movement in a direction differentfrom the image capture direction. The accumulated information is storedin the storage unit 18.

Next, the detection unit 15 detects a reflection in image data to beprocessed on the basis of the accumulated information. The detectionunit 15 then removes the detected reflection from the image to beprocessed.

The recognition unit 16 recognizes a recognition target. In thisembodiment, the recognition unit 16 recognizes a lane line. A knownmethod is used for the process for recognizing a lane line.Alternatively, the recognition target may be another target such as aguardrail.

The calculation unit 17 calculates various values on the basis of aresult of the recognition process. In this embodiment, the calculationunit 17 calculates the relative distance between the vehicle 1 and thecenter of a road on the basis of the position of the recognized laneline in an image. The relative distance is stored in the storage unit 18as a position Xt of the vehicle 1 at a certain time t.

The storage unit 18 stores the accumulated information, informationregarding the positions of the vehicle 1, and information regardingvarious thresholds. The accumulated information is, for example, storedin a table format. The information regarding the positions of thevehicle 1 includes at least information regarding the positions of thevehicle 1 at two latest times, and may be sequentially updated as a newposition of the vehicle 1 is calculated. Various thresholds will bedescribed later.

FIG. 3 is a diagram illustrating an example of a data table for theaccumulated information. Appearance strength Qi is associated with aposition i (Xj, Yk) of each pixel and stored. In FIG. 3, for example, aposition (0, 0) has an appearance strength Q (0, 0). The appearancestrength Q is updated by the detection unit 15. A reflection is detectedon the basis of the appearance strength Qi at each position i (Xj, Yk).

Now, a process for accumulating features performed by the detection unit15 will be described with reference to FIG. 4. FIG. 4 is a diagramillustrating the process for accumulating features. In FIG. 4, assumethat it is determined at a time t1 that there has been no lateralmovement of the vehicle 1 and it is determined at times t2 and t3 thatthere has been lateral movement of the vehicle 1. When there has beenlateral movement, the detection unit 15 updates the appearance strengthat a position at which a feature exists to a larger value.

Edge images 31, 32, and 33 are obtained at the times t1, t2, and t3,respectively. The edge images 31, 32, and 33 are generated by executingthe process for extracting edges on the basis of image data receivedfrom the image capture device 11. For example, the edge image 31indicates that edges 311, 312, and 313 have been extracted at the timet1. Although the edges 311 and 312 are edges caused by lane lines andthe edge 313 is an edge caused by a reflection, these edges are notdistinguished from each other in the edge image 31

Next, images 35, 36, and 37 are based on the information accumulated atthe times t1, t2, and t3, respectively. The images 35, 36, and 37 areimages of edges virtually drawn by providing a pixel value correspondingto the appearance strength Qi to a pixel at each position i. That is,since a larger pixel value is provided as the appearance strength Qibecomes larger, a thicker edge is drawn.

In addition, the information accumulated at each time t is informationobtained by accumulating features in an image at each time t in theinformation accumulated until a previous time (time t-1). For example,the information accumulated at the time t3 is obtained by accumulatingfeatures regarding edges in the image at the time t3 in the informationaccumulated until the time t2.

Here, at the time t1 (there has been no lateral movement), the detectionunit 15 does not accumulate information regarding features in the imageat the time t1 in the accumulated information. That is, with respect tothe position of each pixel included in the edges 311, 312, and 313, theappearance strength Q is not updated to a larger value. When theappearance strength at each position is zero at the time t1, theappearance strength remains at zero.

Next, at the time t2 (there has been lateral movement), the detectionunit 15 accumulates information regarding features in the image at thetime t2 in the accumulated information. That is, a certain value isadded to the appearance strengths Q at positions corresponding to edges321, 322, and 323. As indicated by the image 36, at the time t2, edges361, 362, and 363 are drawn at the positions corresponding to the edges321, 322, and 323, respectively. Broken lines in the edge image 32virtually indicate the positions of the edges at the time t1. That is,when the vehicle 1 laterally moves, the edges caused by the lane linesmove to positions different from those at the time t1.

Next, at the time t3 (there has been lateral movement), the detectionunit 15 accumulates information regarding features in the image at thetime t3 in the accumulated information. That is, the certain value isadded to the appearance strengths Q at the positions of pixelscorresponding to edges 331, 332, and 333. As indicated by the image 37,at the time t3, edges 371, 372, and 373 are drawn at the positionscorresponding to the edges 331, 332, and 333, respectively.

Here, with respect to the position corresponding to the edge 333, since,at the time t3, the certain value is further added to the appearancestrength Q to which the certain value has been added at the time 2, theedge 373 in the image 37 at the time t3 is thicker than the edge 363 inthe image 36. On the other hand, the edges 321 and 322 at the time t2are not extracted at the same positions at the time 3. Therefore, thecertain value is not added to the appearance strengths Q for the edges361 and 362 in the image 36.

In this embodiment, a certain value is subtracted from the appearancestrength Q at a position at which a feature does not exist wheninformation regarding the feature is accumulated. Therefore, even at aposition at which the certain value has been added to the appearancestrength Q because a feature has been extracted in the past, theappearance strength Q is updated to a smaller value if a feature doesnot exist thereafter. However, the minimum value of the appearancestrength Q is 0.

Here, when there has been no movement in a direction different from theimage capture direction, the detection unit 15 does not execute theprocess for accumulating information regarding a feature. However, areflection might disappear when the conditions inside the vehicle 1 havechanged. For example, a reflection might disappear when sunlight isblocked by an object such as when the vehicle 1 is passing under a landbridge or when the vehicle 1 has entered a building.

When a reflection has disappeared, a process for recognizing thedisappearance of the reflection and immediately reducing the appearancestrength is effective. That is, as described later, because when areflection is to be detected, a position at which the appearancestrength is large is determined to be a position at which a reflectionis occurring, it is undesirably determined that the reflection is stilloccurring at the position even after the reflection disappears if theappearance strength remains large.

Therefore, even when there has been no movement in a direction differentfrom the image capture direction, the process for reducing theappearance strength may be performed using information regarding thedisappeared feature. That is, the detection unit 15 executes a processfor checking whether or not features have disappeared in a latest image.

FIG. 5 is a diagram illustrating a process for checking features. InFIG. 5, assume that it is determined at the time t1 that there has beenlateral movement and it is determined at the times t2 and t3 that therehas been no lateral movement. In the checking process, for example, thedetection unit 15 updates the appearance strength to a smaller value ata position at which the appearance strength is equal to or larger than acertain value when a feature has disappeared in the latest image.

Edge images 41, 42, and 43 are obtained at the times t1, t2, and t3,respectively. Images 45, 46, and 47 are based on the informationaccumulated at the times t1, t2, and t3, respectively. At the time t1(there has been lateral movement), information is accumulated on thebasis of the edge image 41. An image of the accumulated information isthe image 45.

Here, at the time t2 (there has been no lateral movement), first, thedetection unit 15 refers to the information accumulated until the timet1. The detection unit 15 then checks with the image at the time t2whether or not there is an edge at each position at which the appearancestrength Q is equal to or larger than the certain value. That is, thedetection unit 15 determines whether or not there is an edge in the edgeimage 42 at the position of each pixel configuring an edge in the image45.

If there is an edge, the appearance strength Q remains the same. Thatis, since edges are located at the same positions in the edge image 42and the image 45, the information accumulated at the time t2 is the sameas the information accumulated at the time t1. That is, the images 45and 46 become the same.

Next, assume that, at the time t3 (there has been no lateral movement),an edge 423 that existed at the time t2 has disappeared as indicated bythe edge image 43. The detection unit 15 refers to the informationaccumulated until the time t2 and checks whether or not there is an edgein the image at the time t3 at each position at which the appearancestrength Q is equal to or larger than the certain value.

An edge 463 in the image 46 does not exist in the edge image 43 at thetime t3. Therefore, at the time t3, the certain value is subtracted fromthe appearance strength Q at the corresponding position. When thecertain value has been subtracted, the image of the accumulatedinformation becomes the image 47 at the time t3. The value to be addedand the value to be subtracted do not have to be the same, but in FIGS.4 and 5, the same value is used for convenience of description.

When there has been no movement in a direction different from the imagecapture direction, the accumulation process for updating the appearancestrength to a larger value is not performed. This is because although itbecomes more likely to recognize a reflection as the appearance strengthbecomes larger, an accumulation process using an image captured under acondition under which it is difficult to distinguish a reflection fromother features might result in incorrect removal.

Next, a procedure of image processing according to this embodiment willbe described. FIGS. 6A and 6B are flowcharts illustrating the imageprocessing. First, the obtaining unit 13 obtains image data from theimage capture device 11 (Op. 1). The obtained image data is image dataregarding an image captured at the time t. The obtaining unit 13 thencalculates a feature value Mi of each pixel on the basis of the obtainedimage data (OP. 2). In this embodiment, the feature value Mi is the edgestrength of each pixel.

Next, the determination unit 14 determines whether or not informationregarding the positions of the vehicle 1 at two points of time in thepast exists (Op. 3). For example, the determination unit 14 refers tothe storage unit 18 and determines whether or not the storage unit 18stores a position X_t-1 of the vehicle 1 at a previous time and aposition X_t-2 of the vehicle 1 at a time before the previous time.Although the positions of the vehicle 1 at the previous time of thecurrent time t and the time before the previous time are used here, theprocessing of the determination is not limited to this.

For example, when the storage unit 18 stores the position of the vehicle1 at each time calculated in the past, the determination unit 14 maydetermine whether or not the storage unit 18 stores the position of thevehicle 1 at the previous time and the position of the vehicle 1 at atime n times before the current time t. Alternatively, when the storageunit 18 stores a position X_base of the vehicle 1 that serves as areference and the position X_t-1 at the previous time, the determinationunit 14 may determine whether or not the storage unit 18 stores theposition X_base and the position X_t-1. If it is determined in laterprocessing that the vehicle 1 has moved in a direction different fromthe image capture direction, the position of the vehicle 1 at that timeis stored as the position X_base. The position X_base is updated eachtime it has been determined that the vehicle 1 has moved in a differentdirection.

If information regarding the positions of the vehicle 1 at two points oftime in the past exists (YES in Op. 3), the determination unit 14calculates a movement distance ΔX (Op. 4). The movement distance is thedistance of movement of the vehicle 1 in a direction different from theimage capture direction. That is, in this embodiment, the movementdistance is the distance of lateral movement. In this embodiment, thedistance ΔX of lateral movement at the previous time is used. This isbecause, in consideration of the frame intervals of the image capturedevice 11, information at the previous time may be sufficiently used forthe determination as to lateral movement at the time t. However,real-time lateral movement may be detected using the steering anglesensor or the like, instead.

Next, the determination unit 14 determines whether or not the movementdistance ΔX is larger than a threshold Tx (Op. 5). If the movementdistance ΔX is larger than the threshold Tx, it is determined that therehas been lateral movement. If the movement distance ΔX is smaller thanor equal to the threshold Tx, it is determined that there has been nolateral movement. The threshold Tx is determined on the basis ofinternal parameters such as the resolution of the image capture device11 and the installation conditions. For example, in this embodiment, thethreshold Tx is 200 mm.

If the movement distance ΔX is larger than the threshold Tx (YES in Op.5), that is, if there has been lateral movement, the detection unit 15executes the process for accumulating features (Op. 6). The accumulationprocess will be described in detail hereinafter. FIG. 7 is a flowchartillustrating the process for accumulating features.

First, the detection unit 15 sets an unprocessed position i as a targetposition (Op. 61). Next, the detection unit 15 obtains the feature valueMi of the target position i in an image at the time t (Op. 62). Next,the detection unit 15 determines whether or not the feature value Mi ofa target pixel is equal to or larger than a threshold Tm1 (Op. 63).

The threshold Tm1 is determined, for example, in accordance with theinternal parameters of the image capture device 11 and an algorithm forextracting a feature. For example, in this embodiment, the edgestrength, which is the feature value, is assumed to range from 0 to 255,and the threshold Tm1 is set to 10.

If the feature value Mi is equal to or larger than the threshold Tm1(YES in Op. 63), the detection unit 15 refers to the accumulatedinformation and adds a certain value Vp to the appearance strength Qicorresponding to the target position i (Op. 64). On the other hand, ifthe feature value Mi is smaller than the threshold Tm1 (NO in Op. 63),the detection unit 15 refers to the accumulated information andsubtracts a certain value Vn1 from the appearance strength Qicorresponding to the target position i (Op. 65). However, if theappearance strength Qi becomes smaller than zero as a result of thesubtraction, the appearance strength Qi becomes zero.

That is, if there is a feature at the position i in the image at thetime t, the detection unit 15 updates the appearance strength Qi in theaccumulated information to a larger value. On the other hand, if thereis no feature at the position i in the image at the time t, thedetection unit 15 updates the appearance strength Q in the accumulatedinformation to a smaller value.

Next, the detection unit 15 determines whether or not the processing hasbeen completed for all the positions i (Op. 66). If the processing hasnot been completed (NO in Op. 66), the process returns to Op. 61, andthe same processing is performed for a new position i. On the otherhand, if the processing has been completed (YES in Op. 66), the processfor accumulating features ends. The detection unit 15 then executes aprocess for detecting reflections, which will be described later.

In FIG. 6A, if information regarding the positions of the vehicle 1 attwo points of time in the past does not exist (NO in Op. 3), or if themovement distance ΔX is smaller than or equal to the threshold Tx (NO inOp. 5), the detection unit 15 executes the process for checking features(Op. 7). The process for checking features will be described in detailhereinafter. FIG. 8 is a flowchart illustrating the process for checkingfeatures.

The detection unit 15 sets an unprocessed position i as a targetposition (Op. 71). The detection unit 15 then obtains the appearancestrength Qi at the target position i from the accumulated informationstored in the storage unit 18 (Op. 72). Next, the detection unit 15determines whether or not the appearance strength Qi is equal to orlarger than a threshold Tq1 (Op. 73).

An appropriate value is set to the threshold Tq1 in accordance with thesystem to be applied. The threshold Tq1 is set on the basis of the rangeof values of the appearance strength Qi. In this embodiment, theappearance strength Q ranges from 0 to 255 (Vp is 0.5), and thethreshold Tq1 is 3. Furthermore, for example, when a possibility thatfeatures other than reflections are incorrectly removed is to be furtherreduced, a smaller value may be set.

If the appearance strength Qi is equal to or larger than the thresholdTq1 (YES in Op. 73), the detection unit 15 obtains the feature value Miof a pixel corresponding to the target position i in the image at thetime t1 (Op. 74). The detection unit 15 then determines whether or notthe feature value Mi is smaller than a threshold Tm2 (Op. 75).

That is, if there has been a feature in past images with an appearancestrength equal to or larger than the certain threshold, the detectionunit 15 checks whether or not there is still the feature at the sameposition at the time t. The threshold Tm2 is determined, for example, inaccordance with the internal parameters of the image capture device 11and the algorithm for extracting a feature. For example, the thresholdTm2 may be the same as the threshold Tm1.

If the feature value Mi is smaller than the threshold Tm2 (YES in Op.75), the detection unit 15 subtracts a certain value Vn2 from theappearance strength Qi at the target position i (Op. 76). That is, if apossibility that a reflection has disappeared is detected, the detectionunit 15 reduces the appearance strength Qi at a position at which thereflection is likely to have existed.

Next, the detection unit 15 determines whether or not the processing hasbeen completed for all the positions i (Op. 77). If the processing hasnot been completed (NO in Op. 77), the process returns to Op. 71, andthe same processing is performed for a new position i. On the otherhand, if the processing has been completed (YES in Op. 77), the processfor checking features ends. The detection unit 15 then executes theprocess for detecting reflections, which will be described later.

If the appearance strength Qi is smaller than the threshold Tq1 (NO inOp. 73), or if the feature value Mi is equal to or larger than thethreshold Tm2 (NO in Op. 75), the detection unit 15 executes theprocessing in Op. 77.

Here, the relationship between Vp, Vn1, and Vn2 will be described. Vp isset to a value larger than Vn1 and Vn2. Alternatively, the relationshipbetween Vp, Vn1, and Vn2 is determined in accordance with the system tobe applied. That is, the relationship between Vp, Vn1, and Vn2 isdetermined on the basis of whether suppression of incorrect removal orremoval of reflections takes priority over the other.

For example, when removal of reflections is to take priority, Vp is setto a value larger than Vn1 and Vn2. That is, when an image in whichthere has been lateral movement has been captured, the appearancestrength Qi reaches the threshold Tq2, at which a reflection is removed,with a smaller delay.

In this embodiment, for example, Vp is set to a value twice Vn1. Vn1 isappropriately set on the basis of the range of values of the appearancestrength Qi. In addition, for example, Vn2 is set to a value half Vn1.

Next, when the process for accumulating features (Op. 6) or the processfor checking features (Op. 7) has ended, the detection unit 15 executesthe process for detecting reflections (Op. 8). The process for detectingreflections will be described in detail hereinafter. FIG. 9 is aflowchart illustrating the process for detecting reflections. Theprocess for accumulating features and the process for checking featuresmay be executed at each time or may be executed at certain timeintervals.

First, the detection unit 15 sets an unprocessed position i as a targetposition (Op. 81). Next, the detection unit 15 obtains the appearancestrength Qi at the target position i from the accumulated informationstored in the storage unit 18 (Op. 82). The detection unit 15 thendetermines whether or not the appearance strength Qi is equal to orlarger than the threshold Tq2 (Op. 83).

The threshold Tq2 is determined on the basis of Vp, Vn1, and Vn2. Forexample, when a reflection is to be removed with a smaller delay, thethreshold Tq2 is set to a small value. In this embodiment, the thresholdTq2 is set to a value five times Vp.

If the appearance strength Qi is equal to or larger than the thresholdTq2 (YES in Op. 83), a feature located at the position i in the image atthe time t is detected as a reflection. That is, the detection unit 15sets the feature value Mi of a pixel at the target position i to zero(Op. 84). In other words, the feature caused by the reflection isremoved from the image at the time t. In this embodiment, an edge isremoved by setting the edge strength in an edge image to zero.

In this embodiment, when a reflection has been detected, the reflectionis removed. However, the technique disclosed herein is not limited tothis, and only a result of the detection may be output without removingthe reflection depending on subsequent processing.

Next, the detection unit 15 determines whether or not the processing hasbeen completed for all the positions i (Op. 85). The processing in Op.85 is also executed if the appearance strength Qi is smaller than thethreshold Tq2 (NO in Op. 83).

If the processing has not been completed for all the positions i (NO inOp. 85), the process returns to Op. 81, and the same processing isperformed for a new position i. On the other hand, if the processing hasbeen completed (YES in Op. 85), the process for detecting reflectionsends. The detection unit 15 then outputs the image from which detectedreflections have been removed to a processing unit that executes thesubsequent processing.

When the process for detecting reflections (Op. 8) has ended in FIG. 6A,processing in Op. 9 illustrated in FIG. 6B and later is executed. First,the recognition unit 16 recognizes a recognition target in the imagefrom which reflections have been removed (Op. 9). One of various knownalgorithms is applied in accordance with the recognition target. Forexample, the recognition unit 16 recognizes a lane line by applying analgorithm for recognizing a lane line. The recognition unit 16 thenoutputs a result of the recognition to a processing unit in a subsequentstage.

Next, the calculation unit 17 calculates the position Xt of the vehicle1 at the time t on the basis of information regarding the position ofthe lane line and various parameters (Op. 10). A known method is appliedas a method for calculating the position Xt of the vehicle 1 using theposition of the lane line.

For example, the calculation unit 17 models the position of the laneline on the road. The calculation unit 17 then calculates the position Xof the lane line on the road from the position of the lane line in theimage. The following expression is applied:

$x = {{\frac{1}{2}\frac{{cf}^{2}H}{\left( {y + {f \cdot \phi}} \right)}} + {\left( {{k\frac{W}{2}} + X} \right)\frac{\left( {y + {f \cdot \phi}} \right)}{H}} + {f \cdot \theta}}$

x denotes the x-coordinate of the lane line in the image. y denotes they-coordinate of the lane line in the image. f denotes the focal distanceof a lens of the image capture device 11. H denotes the installationheight of the image capture device 11. W denotes the width of the laneline specified in the Road Traffic Law or the like. θ denotes the yawangle of the image capture device 11. φ denotes the pitch angle of thecamera. c denotes the curvature of the road. k is a constant dependingon the direction of the lane line. k is −1 in the case of a left laneline and +1 in the case of a right lane line.

x and y are values obtained from the result of the recognition. f, H, W,and k are predetermined values. By repeatedly performing calculationadopting linear approximation using the above expression, the position Xof the vehicle 1, the yaw angle θ, the pitch angle φ, and the curvaturec of the road are calculated. Although the position X of the vehicle 1is output in this embodiment because only the position X of the vehicle1 is used in subsequent processing, the yaw angle θ, the pitch angle φ,and the curvature c may also be output, instead.

The calculation unit 17 outputs the calculated position Xt of thevehicle 1 to the warning device 19 and stores the position Xt in thestorage unit 18 (Op. 11). For example, when the position Xt of thevehicle 1 indicates that the vehicle 1 is deviating from the trafficlane, the warning device 19 outputs a warning tone through a speaker.

The calculation unit 17 then determines whether or not to end the imageprocessing (Op. 12). If the image processing is to end (YES in Op. 12),the processing ends. On the other hand, if the image processing is tocontinue (NO in Op. 12), the processing returns to Op. 1, and theprocessing is performed for a new image. For example, when an engine hasstopped, the image processing ends.

As described above, in a technique for removing reflections from animage, the image processing device 12 may detect reflections from animage at a time when the vehicle 1 has moved in a direction differentfrom the image capture direction. That is, features remaining at thesame positions in images before and after the vehicle 1 moves in adirection different from the image capture direction are detected asreflections. Therefore, the image processing device 12 may accuratelydistinguish reflections and objects other than the reflections from eachother.

FIG. 10 is a diagram illustrating an example of the hardwareconfiguration of a computer 200 that functions as the image processingdevice 12. The computer 200 may also function as the driving supportapparatus 10 along with a camera 207.

The computer 200 is connected to the camera 207 and a speaker 208. Thecamera 207 is an example of the image capture device 11. The speaker 208is a device that outputs a warning tone under control of the warningdevice 19 when the warning device 19 issues a warning.

The computer 200 includes a central processing unit (CPU) 201, aread-only memory (ROM) 202, a random-access memory (RAM) 203, a harddisk drive (HDD) 204, and a communication device 205. The computer 200may further include a medium reading device.

These components are connected to one another through a bus 206. Thesecomponents may transmit and receive data to and from one another undercontrol of the CPU 201.

A program in which the method for processing an image illustrated inFIGS. 6A, 6B, and 7 is described is recorded on a recording medium thatmay be read by the computer 200. The recording medium that may be readby the computer 200 may be a magnetic recording device, an optical disk,a magneto-optical recording medium, a semiconductor memory, or the like.The magnetic recording device may be an HDD, a flexible disk (FD), amagnetic tape (MT), or the like.

The optical disk may be a digital versatile disc (DVD), a DVD-RAM, acompact disc read-only memory (CD-ROM), a CD-recordable (CD-R), aCD-rewritable (CD-RW), or the like. The magneto-optical recording mediummay be a magneto-optical (MO) disk or the like. When the program is tobe distributed, for example, portable recording media such as DVDs orCD-ROMs on which the program is recorded are sold.

For example, the medium reading device reads the program from arecording medium on which the image processing program is recorded. TheCPU 201 stores the read program in the HDD 204. Alternatively, variousprograms may be stored in the ROM 202 or the RAM 203 that may beaccessed by the CPU 201.

The CPU 201 is a central processing unit that controls the entireoperation of the computer 200. By reading various programs from the HDD204 and executing various programs, the CPU 201 functions as theobtaining unit 13, the determination unit 14, the detection unit 15, therecognition unit 16, and the calculation unit 17.

The HDD 204 also functions as the storage unit 18 under control of theCPU 201. As with the programs, information in the storage unit 18 may bestored in the ROM 202 or the RAM 203 that may be accessed by the CPU201, instead. The RAM 203 also stores information temporarily generatedduring processes. The communication device 205 transmits and receivesinformation to and from other devices connected through an interface.That is, the communication device 205 may realize part of the functionof the obtaining unit 13.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the principlesof the invention and the concepts contributed by the inventor tofurthering the art, and are to be construed as being without limitationto such specifically recited examples and conditions, nor does theorganization of such examples in the specification relate to a showingof the superiority and inferiority of the invention. Although theembodiments of the present inventions have been described in detail, itshould be understood that the various changes, substitutions, andalterations could be made hereto without departing from the spirit andscope of the invention.

What is claimed is:
 1. An image processing device comprising: a memory;and a processor coupled to the memory and configured to: obtain imagedata regarding an image captured by an image capture device that movesalong with a moving body and of which image capture direction is acertain traveling direction of the moving body, and detect a featurecaused by a reflection from features based on information regardingpositions of the features in the image data at a time when the movingbody has moved in a direction different from the image capturedirection.
 2. The image processing device according to claim 1, whereinthe processor is further configured to update appearance strengths ofthe information regarding the positions of the features in a pluralityof pieces of image data at a time when the moving body has moved in adirection different from the image capture direction, and detect thefeature caused by the reflection at a position at which the appearancestrength is equal to or larger than a threshold, and wherein theappearance strengths depend on frequencies of appearances of thefeatures.
 3. The image processing device according to claim 1, whereinthe certain traveling direction is a forward direction of the movingbody, and wherein the different direction is a lateral direction.
 4. Theimage processing device according to claim 1, wherein the processor isfurther configured to remove the feature caused by the reflection fromthe image data, and execute a process for recognizing a target in theimage data from which the reflection has been removed.
 5. The imageprocessing device according to claim 4, wherein the target is a lineconfiguring a traffic lane, and wherein the moving body is a vehicle. 6.A method for processing an image executed by a computer, the methodcomprising: obtaining image data regarding an image captured by an imagecapture device that moves along with a moving body and of which imagecapture direction is a certain traveling direction of the moving body;and detecting a feature caused by a reflection from features based oninformation regarding positions of the features in the image data at atime when the moving body has moved in a direction different from theimage capture direction.
 7. The method according to claim 6, furthercomprising: updating appearance strengths of the information regardingthe positions of the features in a plurality of pieces of image data ata time when the moving body has moved in a direction different from theimage capture direction; and detecting the feature caused by thereflection at a position at which the appearance strength is equal to orlarger than a threshold, and wherein the appearance strengths depend onfrequencies of appearances of the features.
 8. The method according toclaim 6, wherein the certain traveling direction is a forward directionof the moving body, and wherein the different direction is a lateraldirection.
 9. The method according to claim 6, further comprising:removing the feature caused by the reflection from the image data; andexecuting a process for recognizing a target in the image data fromwhich the reflection has been removed.
 10. The method according to claim9, wherein the target is a line configuring a traffic lane, and whereinthe moving body is a vehicle.
 11. A device, comprising: a memory; and aprocessor coupled to the memory and configured to: obtain image dataregarding an image captured by an image capture device that moves alongwith a moving body and of which image capture direction is a certaintraveling direction of the moving body, identify a feature from theimage data based on information regarding positions of features in theimage data at a time when the moving body has moved in a directiondifferent from the image capture direction, when a value assigned to thefeature over a time period of movement of the moving body is below athreshold, and issue a warning based on the feature, the warning warns adeviation of the moving body from a lane.
 12. The device according toclaim 11, wherein the value indicates appearance strength of the featurebased on frequency of appearances of the feature at a position duringthe movement in the direction different from the image capturedirection.